Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MARC21/XML support #659

Closed
wants to merge 81 commits into from
Closed

Add MARC21/XML support #659

wants to merge 81 commits into from

Conversation

jimjonesbr
Copy link
Contributor

Add functions ST_AsMARC21 and ST_GeomFromMARC21 with their XML documentation files and regression tests.

jimjonesbr and others added 30 commits January 7, 2022 15:33
- ST_GeomFromMARC21: extracts geometries from MARC21 documents
- ST_AsMARC21: generates MARC21 documents from geometries
postgis/lwgeom_in_marc21.c Outdated Show resolved Hide resolved
@jimjonesbr jimjonesbr requested a review from strk February 9, 2022 12:46
@robe2
Copy link
Member

robe2 commented Feb 13, 2022

@pramsey @jimjonesbr Are we ready to push this thru now?

@jimjonesbr
Copy link
Contributor Author

jimjonesbr commented Feb 13, 2022

Are we ready to push this thru now?

@robe2 I've already addressed the points made by @pramsey and @strk, except the suggestion of @pramsey to use regex in a specific function. Nevertheless, I refactored it to make things more readable and avoid memcpy calls (I hope it is more palatable now: #11bedb6). I was waiting for the other reviewers to make any further change to the code.

postgis/lwgeom_in_marc21.c Outdated Show resolved Hide resolved
postgis/lwgeom_in_marc21.c Outdated Show resolved Hide resolved
postgis/lwgeom_in_marc21.c Outdated Show resolved Hide resolved
postgis/postgis.sql.in Outdated Show resolved Hide resolved
jimjonesbr and others added 2 commits February 16, 2022 00:14
- adds null check in the literal input in 'is_literal_valid
- removes unnecessary lwgeom_free(geom) in 'ST_GeomFromMARC21'
- changes '$libdir/postgis-3' to 'MODULE_PATH' in PLpg/SQL function
- changes 'PG_GETARG_GSERIALIZED_P_COPY' to 'PG_GETARG_GSERIALIZED_P'
  in 'ST_AsMARC21'
- changes sprintf calls to snprintf in 'corner_to_subfield_sb'
@robe2
Copy link
Member

robe2 commented Feb 17, 2022

Anyone else have anything to say before I commit this.

I'm going to ignore the cloud drone failure since the PG 11 run passed and looked like the rest was just halted for some reason.

@pramsey
Copy link
Member

pramsey commented Feb 17, 2022

This has been squashed into master now.

@pramsey pramsey closed this Feb 17, 2022
@strk
Copy link
Member

strk commented Feb 17, 2022

Tests are failing on 32bit architecture (rounding?): https://gitlab.com/postgis/postgis/-/pipelines/473567768

@jimjonesbr
Copy link
Contributor Author

Tests are failing on 32bit architecture (rounding?): https://gitlab.com/postgis/postgis/-/pipelines/473567768

@strk Yes, it's a rounding problem. It seems I accidentally skipped the roundoff ST_AsTest(geom, 5) in a few test cases

@robe2
Copy link
Member

robe2 commented Feb 25, 2022

@jimjonesbr Still getting slight rounding issue on windows - https://trac.osgeo.org/postgis/ticket/5104

 ./regress/core/out_marc21 .. failed (diff expected obtained: /projects/postgis/tmp/3.3.0dev_pg11_geos3.11_gdal3.3.3w64/test_27_diff)
> -----------------------------------------------------------------------------
> --- ./regress/core/out_marc21_expected	2022-02-17 19:39:49.495811700 -0500
> +++ /projects/postgis/tmp/3.3.0dev_pg11_geos3.11_gdal3.3.3w64/test_27_out	2022-02-25 11:51:05.676040100 -0500
> @@ -781,7 +781,7 @@
>  polygon_14|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E126.416667</subfield><subfield code="e">E129.258333</subfield><subfield code="f">N036.127222</subfield><subfield code="g">N035.033333</subfield></datafield></record>
>  polygon_15|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E12625.000000</subfield><subfield code="e">E12915.500000</subfield><subfield code="f">N03607.633333</subfield><subfield code="g">N03503.000000</subfield></datafield></record>
>  polygon_16|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E1262500.000000</subfield><subfield code="e">E1291530.000000</subfield><subfield code="f">N0360738.000000</subfield><subfield code="g">N0350200.000000</subfield></datafield></record>
> -polygon_17|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E126.416666666666671403618238400667905807495117</subfield><subfield code="e">E129.258333333333325754210818558931350708007812</subfield><subfield code="f">N036.127222222222222569598670816048979759216309</subfield><subfield code="g">N035.033333333333331438552704639732837677001953</subfield></datafield></record>
> +polygon_17|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E126.416666666666671403618238400667905807495117</subfield><subfield code="e">E129.258333333333325754210818558931350708007813</subfield><subfield code="f">N036.127222222222222569598670816048979759216309</subfield><subfield code="g">N035.033333333333331438552704639732837677001953</subfield></datafield></record>
>  polygon_18|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E12625.000000000000284217094304040074348449707031250</subfield><subfield code="e">E12915.499999999999545252649113535881042480468750000</subfield><subfield code="f">N03607.633333333333354175920248962938785552978515625</subfield><subfield code="g">N03502.999999999999886313162278383970260620117187500</subfield></datafield></record>
>  polygon_19|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E1262500.0000000000170530256582424044609069824218750</subfield><subfield code="e">E1291529.9999999999727151589468121528625488281250000</subfield><subfield code="f">N0360738.0000000000012505552149377763271331787109375</subfield><subfield code="g">N0350200.0000000000068212102632969617843627929687500</subfield></datafield></record>
>  polygon_20|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">1262500</subfield><subfield code="e">1291530</subfield><subfield code="f">0360738</subfield><subfield code="g">0350200</subfield></datafield></record>

E129.258333333333325754210818558931350708007812

vs.

E129.258333333333325754210818558931350708007813

@jimjonesbr
Copy link
Contributor Author

@jimjonesbr Still getting slight rounding issue on windows - https://trac.osgeo.org/postgis/ticket/5104

 ./regress/core/out_marc21 .. failed (diff expected obtained: /projects/postgis/tmp/3.3.0dev_pg11_geos3.11_gdal3.3.3w64/test_27_diff)
> -----------------------------------------------------------------------------
> --- ./regress/core/out_marc21_expected	2022-02-17 19:39:49.495811700 -0500
> +++ /projects/postgis/tmp/3.3.0dev_pg11_geos3.11_gdal3.3.3w64/test_27_out	2022-02-25 11:51:05.676040100 -0500
> @@ -781,7 +781,7 @@
>  polygon_14|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E126.416667</subfield><subfield code="e">E129.258333</subfield><subfield code="f">N036.127222</subfield><subfield code="g">N035.033333</subfield></datafield></record>
>  polygon_15|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E12625.000000</subfield><subfield code="e">E12915.500000</subfield><subfield code="f">N03607.633333</subfield><subfield code="g">N03503.000000</subfield></datafield></record>
>  polygon_16|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E1262500.000000</subfield><subfield code="e">E1291530.000000</subfield><subfield code="f">N0360738.000000</subfield><subfield code="g">N0350200.000000</subfield></datafield></record>
> -polygon_17|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E126.416666666666671403618238400667905807495117</subfield><subfield code="e">E129.258333333333325754210818558931350708007812</subfield><subfield code="f">N036.127222222222222569598670816048979759216309</subfield><subfield code="g">N035.033333333333331438552704639732837677001953</subfield></datafield></record>
> +polygon_17|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E126.416666666666671403618238400667905807495117</subfield><subfield code="e">E129.258333333333325754210818558931350708007813</subfield><subfield code="f">N036.127222222222222569598670816048979759216309</subfield><subfield code="g">N035.033333333333331438552704639732837677001953</subfield></datafield></record>
>  polygon_18|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E12625.000000000000284217094304040074348449707031250</subfield><subfield code="e">E12915.499999999999545252649113535881042480468750000</subfield><subfield code="f">N03607.633333333333354175920248962938785552978515625</subfield><subfield code="g">N03502.999999999999886313162278383970260620117187500</subfield></datafield></record>
>  polygon_19|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">E1262500.0000000000170530256582424044609069824218750</subfield><subfield code="e">E1291529.9999999999727151589468121528625488281250000</subfield><subfield code="f">N0360738.0000000000012505552149377763271331787109375</subfield><subfield code="g">N0350200.0000000000068212102632969617843627929687500</subfield></datafield></record>
>  polygon_20|<record xmlns="http://www.loc.gov/MARC21/slim"><datafield tag="034" ind1="1" ind2=" "><subfield code="a">a</subfield><subfield code="d">1262500</subfield><subfield code="e">1291530</subfield><subfield code="f">0360738</subfield><subfield code="g">0350200</subfield></datafield></record>

E129.258333333333325754210818558931350708007812

vs.

E129.258333333333325754210818558931350708007813

Hi @robe2. These tests were originally designed to make sure one can create coordinates in marc with as much decimal places as wanted... but now I see it creates too much trouble across architectures and platforms. I will send a PR with an alternative test.

@jimjonesbr
Copy link
Contributor Author

Hi @robe2
Where can I find the online documentation for the MARC21 functions? Should it be something like https://postgis.net/docs/ST_AsMARC21.html ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants